Method of making a digital camera image of a first scene with a superimposed second scene

ABSTRACT

A method of making an image in a digital camera comprises capturing a digital image of a scene into which the camera user is to be inserted, and superimposing a symbol (subject locator) onto the scene image representing at least a part of a human subject. The subject locator is scaled to a desired size and moved to a desired position relative to the scene image. Next a digital image of the user is captured, and at least the part of the user image represented by the subject locator is extracted. The part of the user image represented by the subject locator is scaled (before or after extraction) to substantially the same size as the subject locator and inserted into the first image at the position of the subject locator. In a second embodiment the method comprises displaying a preview image of a scene into which the camera user is to be inserted, and superimposing the subject locator on the preview image. The subject locator is scaled to a desired size and moved to a desired position relative to the edges of the preview image. Face detection detects the camera user entering the scene displayed by the preview image. The preview image is scaled and panned to bring the part of the preview image represented by the subject locator to substantially the same size and position as the subject locator. Finally, a digital image of the scene is captured.

PRIORITY

This application claims benefit under 35 U.S.C. § 120 as a Continuationof application Ser. No. 13/753,426, filed on Jan. 29, 2013, which claimsthe benefit under 35 U.S.C. § 120 as a continuation of U.S. patentapplication Ser. No. 12/572,930, filed on Oct. 2, 2009, now U.S. Pat.No. 8,379,917, the entire contents of both of which are herebyincorporated by reference for all purposes as if fully set forth herein.The applicant(s) hereby rescind any disclaimer of claim scope in theparent application(s) or the prosecution history thereof and advise theUSPTO that the claims in this application may be broader than any claimin the parent application(s).

BACKGROUND

A disadvantage with conventional digital cameras is that the camerauser, i.e. the photographer, is located on the opposite side of thecamera to the scene being photographed, so that he is automaticallyexcluded from the scene. Self-timers which set a delay between pressingthe shutter button and releasing the shutter allow the user to moveround to the front of the camera in time to appear in the scene.However, the user has to position himself in the scene by guesswork andhas no accurate control as to his position or size in the scene.

US Patent Application Publication No. US 2006/0125928 discloses adigital camera having forward and rear facing lenses, so that an imageof the user can be taken at the same time as the image of the scene. Theimage of the user is then “associated” with the image of the scene.However, such association does not provide a natural integration of theuser into the scene.

SUMMARY

In a first embodiment, a method of making an image in a digital camerais provided, comprising capturing a digital image of a scene into whichthe camera user is to be inserted, and superimposing a symbol (subjectlocator) onto the scene image representing at least a part of a humansubject. The subject locator is scaled to a desired size and moved to adesired position relative to the scene image. Next a digital image ofthe user is captured, and at least the part of the user imagerepresented by the subject locator is extracted. The part of the userimage represented by the subject locator is scaled (before or afterextraction) to substantially the same size as the subject locator andinserted into the first image at the position of the subject locator.

In a second embodiment, a further method of making an image in a digitalcamera is provided, comprising displaying a preview image of a sceneinto which the camera user is to be inserted, and superimposing thesubject locator on the preview image. The subject locator is scaled to adesired size and moved to a desired position relative to the edges ofthe preview image. The camera user is detected entering the scenedisplayed by the preview image, and the preview image is scaled andpanned to bring the part of the preview image represented by the subjectlocator to substantially the same size and position as the subjectlocator. Finally, a digital image of the scene is captured.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will now be described by way of examplewith reference to the accompanying drawings, in which:

FIG. 1 is a block diagram of a digital camera operating in accordancewith an embodiment of the present invention.

FIG. 2 is a flow diagram of the steps performed by software in thecamera of FIG. 1 in a first embodiment of the invention.

FIGS. 3.1 to 3.4 are schematic diagrams illustrating the operation ofthe first embodiment.

FIG. 4 is a flow diagram of the steps performed by software in thecamera of FIG. 1 in a second embodiment of the invention.

FIGS. 5.1 to 5.3 are schematic diagrams illustrating the operation ofthe second embodiment.

DETAILED DESCRIPTION OF THE EMBODIMENTS

FIG. 1 is a block diagram of a digital image acquisition device 20 whichmay be a portable digital camera per se or a digital camera incorporatedinto a cell phone (in the latter case only the camera components of thephone are shown). The device includes a processor 120. It can beappreciated that many of the processes implemented in the digital cameramay be implemented in or controlled by software operating in amicroprocessor, central processing unit, controller, digital signalprocessor and/or an application specific integrated circuit,collectively depicted as processor 120. Generically, all user interfaceand control of peripheral components such as buttons and display iscontrolled by a microcontroller 122. The processor 120, in response to auser input at 122, such as half pressing a shutter button (pre-capturemode 32), initiates and controls the digital photographic process.Ambient light exposure is monitored using light sensor 40 in order toautomatically determine if a flash is to be used. A distance to thesubject is determined using a focus component 50, which controls azoomable main lens system 62 on the front of the camera to focus animage of an external scene onto an image capture component 60 within thecamera. If a flash is to be used, processor 120 causes the flash 70 togenerate a photographic flash in substantial coincidence with therecording of the image by image capture component 60 upon fulldepression of the shutter button. The image capture component 60digitally records the image in colour. The image capture componentpreferably includes a CCD (charge coupled device) or CMOS to facilitatedigital recording. The flash may be selectively generated either inresponse to the light sensor 40 or a manual input 72 from the user ofthe camera. The high resolution image recorded by image capturecomponent 60 is stored in an image store 80 which may comprise computermemory such a dynamic random access memory or a non-volatile memory. Thecamera is equipped with a display screen 100, such as an LCD, forpreview and post-view of images.

In the case of preview images which are generated in the pre-capturemode 32 with the shutter button half-pressed, the display 100 can assistthe user in composing the image, as well as being used to determinefocusing and exposure. Temporary storage 82 is used to store one or moreof the preview images and can be part of the image store 80 or aseparate component. The preview image is preferably generated by theimage capture component 60. For speed and memory efficiency reasons,preview images preferably have a lower pixel resolution than the mainimage taken when the shutter button is fully depressed, and aregenerated by sub-sampling a raw captured image using software 124 whichcan be part of the general processor 120 or dedicated hardware orcombination thereof. Depending on the settings of this hardwaresubsystem, the pre-acquisition image processing may satisfy somepredetermined test criteria prior to storing a preview image. Such testcriteria may be chronological, such as to constantly replace theprevious saved preview image with a new captured preview image every 0.5seconds during the pre-capture mode 32, until the high resolution mainimage is captured by full depression of the shutter button. Moresophisticated criteria may involve analysis of the preview imagecontent, for example, testing the image for changes, before decidingwhether the new preview image should replace a previously saved image.Other criteria may be based on image analysis such as the sharpness, ormetadata analysis such as the exposure condition, whether a flash isgoing to happen, and/or the distance to the subject.

If test criteria are not met, the camera continues by capturing the nextpreview image without saving the current one. The process continuesuntil the final high resolution main image is acquired and saved byfully depressing the shutter button.

Where multiple preview images can be saved, a new preview image will beplaced on a chronological First In First Out (FIFO) stack, until theuser takes the final picture. The reason for storing multiple previewimages is that the last preview image, or any single preview image, maynot be the best reference image for comparison with the final highresolution image in, for example, a red-eye correction process or, inthe present embodiment, mid-shot mode processing. By storing multipleimages, a better reference image can be achieved, and a closer alignmentbetween the preview and the final captured image can be achieved in analignment stage discussed later.

The camera is also able to capture and store in the temporary storage 82one or more low resolution post-view images. Post-view images are lowresolution images essentially the same as preview images, except thatthey occur after the main high resolution image is captured.

In addition to the zoomable main lens system 62, the camera includes azoomable subsidiary lens system 66 and corresponding image capturecomponent 68. In a cell phone the subsidiary lens system 66 normallyfaces rearwardly towards a user holding the phone, that is, in theopposite direction to the forwardly facing front lens system 62. Thisallows the user to enter into a video phone call with a remote partywhile holding the phone in a natural manner. The components allowingvideo calling are not relevant to the present invention and are notshown. The subsidiary lens system 66 may be focusable, using a focuscomponent 64, or have a fixed focus in which case the focus component 64would be omitted. A user input 84 allows the user to select either oneof the lens systems for use, the same processing circuitry, as shown inFIG. 1, being used for both except that in this embodiment arearward-facing flash, corresponding to the forward-facing flash 70, isomitted.

The camera includes a “User Composite Mode” which can be selected by auser input 30 at any time that a user wishes to be inserted into a sceneimaged by the front lens system 62 and currently previewed on the cameradisplay screen 100. FIG. 2 is a flow diagram of the steps performed bysoftware in the camera of FIG. 1 when User Composite Mode is selected ina first embodiment of the invention. Where a user input is required forany particular step, the existing camera controls may be programmed forthis purpose.

Step 200: In response to full depression of the shutter button, a firststill image 300 (FIG. 3.1) of the scene imaged by the front lens 62 onthe component 60 is captured. The first image 300 is displayed on thescreen 100.

Step 202: Foreground/background separation on the image 300 isoptionally performed using techniques described in, for example,International Patent Application No.'s. PCT/EP2006/008229 (FN119) andPCT/EP2006/005109 (FN122). The separation data is stored for use in step208.

Step 204: In response to user input, a subject locator 302 (FIG. 3.2) isgenerated and superimposed on the displayed image 300. The subjectlocator 302 is a symbol representing all or part of a human subject. Inthe present case the subject locator is a simplified outline of the headand body of a human subject. The subject locator may be available inseveral different profiles corresponding to, e.g., head and shoulders,mid-shot or full length, in which case the user selects the desired one.The subject locator 302 shown in FIG. 3.2 is assumed to be a full lengthprofile.

Step 206: In response to user input, the subject locator 302 is shiftedrelative to the image frame defined by the edges 303 of the displayscreen 100 to place the subject locator at a desired position relativeto the still image 300. The subject locator may also be zoomed (i.e.scaled up or down) to a desired size relative to the image frame. Aconventional four-way directional menu control may be used to shift thesubject locator, and a conventional manual zoom control may be used tozoom the subject locator, both controls being programmed in UserComposite Mode for those purposes.

Step 208: If step 202 was performed, the user also selects, in a casewhere the subject locator 302 partly overlaps the foreground of theimage 300, whether the user is to be inserted in front of or behind theforeground of the image 300.

Step 210: Once selections in step 208 are confirmed, the camera switchesto preview mode of the image seen through the rear lens 66, i.e. animage of the user.

Step 212: In response to full depression of the shutter button, a secondstill image 304 (FIG. 3.3) of the user imaged by the rear lens 66 on thecomponent 68 is captured. The second image 304 is displayed on thescreen 100 for confirmation by the user. If not confirmed, one or morefurther images may be captured until the user is satisfied with thecaptured image 304.

Step 214: Upon confirmation, the software performs face detection and/orforeground/background separation on the second image 304 to locate theuser's face and body 306, or as much as is captured in the image 304.Face detection may use techniques described in, for example,International Patent Application No. PCT/EP2007/005330 (FN143), whileforeground/background separation may use techniques as previouslyreferred to.

Step 216: The software extracts the face and—depending on the profile ofthe selected subject locator—all or part of the user's body from thesecond image 304. For example, if the subject locator were a head andshoulders profile, the software would only extract the head andshoulders of the user. The software then scales the extracted imagecomponent up or down to substantially the same size as the subjectlocator. Alternatively, the scaling could be done by digitally zoomingthe entire second image 304 before extraction of the face and (part ofthe) body.

Step 218: Finally, the image component extracted in step 216 is insertedinto the first image 300 at the position of the subject locator 302 toprovide a composite image 308, FIG. 3.4, in which the inserted imagecomponent replaces the underlying original image data and the subjectlocator is removed. Known blending techniques may be used to smooth thetransition between the inserted image component 306 and the originalscene 300. If steps 202 and 208 were performed in a case where thesubject locator 302 partly overlaps the foreground of the image 300,only that part of the extracted image component overlapping thebackground of the image 300 is inserted into the image 300. In avariation of this step the software could extract all of the face andbody in step 216 and only insert the part corresponding to the selectedsubject locator profile in step 218 (e.g. head and shoulders).

Various modifications of the above embodiment are possible.

The first and second images 300, 304 need not be captured in the orderstated; for example, steps 210 to 214 could be done before steps 200 to208. If desired, bearing mind that in this embodiment the camera hasboth front and rear lens systems, the first and second images could becaptured at substantially the same time. In another modification, one orboth images 300, 304 could be pre-existing images, i.e. captured andstored before the user enters User Composite Mode. In that case, steps200 and 212 would consist of selecting the relevant images from thestored images.

In a case where the camera is not a dual-lens camera, i.e. it has only afront-facing lens 62, the second image 304 could be captured through thefront lens by allowing the user time to move round to the front of thecamera or to turn the camera around to face the user. The second imagecould then either be captured using a timer; or if the camera has asecondary front facing display, through the user manually capturing thesecond image when they are satisfied with the image shown in thesecondary display; or alternatively by automatically capturing asuitable image of the user fitting the profile as described for thesecond embodiment. Further alternatively, the second image 304 could betaken by a third party.

Furthermore, where the camera is provided with a speaker, the softwarecould be arranged to produce audio directions via the speaker in orderto guide the user to a desired location within the scene in order toimprove or replace the scaling referred to in step 216. For example, theuser could be instructed to move left, right, forward or backwardswithin the scene.

In another variation the scaling referred to in step 216 could be donebefore extraction by performing face detection and/orforeground/background separation on a preview of the second image 304 tolocate the user's face and body 306, and then optically zoom the previewso that when the second image is 304 captured the face and body arealready at the correct size for placement at the subject locator 302 inthe image 300.

It is also to be noted that by placing the subject locator 302 in frontof a person in the original scene 300, the user can replace that personin the scene. It is also possible, by having a subject locator profilecorresponding just to a face, to replace a person's face while retainingtheir original clothing, etc.

FIG. 4 is a flow diagram of the steps performed by software in thecamera of FIG. 1 when User Composite Mode is selected in a secondembodiment of the invention. At the commencement of the process it isassumed that the camera is in preview mode and the display 100 isshowing a preview image derived through the front lens system 62, i.e. apreview of a scene into which the user wishes to be inserted. Again,where a user input is required for any particular step, the existingcamera controls may be programmed for this purpose.

Step 400: A face detection algorithm locates and tracks faces (if any)in the displayed preview image 500. In FIG. 5.1 face tracking isindicated by the brackets 502.

Step 402: In response to user input, a subject locator 504 is generatedand superimposed on the displayed preview image 500. As before, thesubject locator may be available in several different profiles, in whichcase the user selects the desired one.

Step 404: In response to user input, the subject locator 504 is shiftedrelative to the image frame defined by the edges 506 of the displayscreen 100 to place the subject locator at a desired position relativeto the preview image 500. The subject locator may also be zoomed to adesired size relative to the image frame.

Step 406: User activates a self-timer button to allow the user to moveround to front of camera and enter the scene.

Step 408: The software detects and tracks an (additional) face 508entering the scene.

Step 410: When the software detects that the additional face 508 hassubstantially stopped moving, or at the expiration of a time period setby the self-timer button, the entire preview image is zoomed (opticallyand/or digitally) and panned (digitally) to bring the image 510 of theuser (or relevant part as determined by the subject locator profile) toa position where it is superimposed on the subject locator 504 with asize substantially the same as that of the subject locator. Note thatthe position of the subject locator 504 is fixed relative to the edges506 of the frame so that panning and zooming the preview imageeffectively moves the entire image relative to the subject locator.

Step 412: When the panning and zooming is complete, the subject locator504 is removed and the scene imaged by the front lens 62 on thecomponent 60 is captured.

In a variation of the above embodiment, where the camera is providedwith a speaker, at step 410, the software is arranged to produce audiodirections via the speaker in order to guide the user to a desiredlocation within the scene. For example, referring to FIGS. 5.2 and 5.3,were the user to enter the scene from the left hand side, he mayposition himself to the left of the subjects already present in thepreview image. In such a case and as a result of the zooming and panningof step 410, it is possible that the captured image may no longerdisplay those subjects, and the preview image would not be substantiallyequal to the image captured. Thus, by guiding the user, for example, byinstructing him to move to the right, an image substantially equal tothat of the preview image can be captured.

While an exemplary drawings and specific embodiments of the presentinvention have been described and illustrated, it is to be understoodthat that the scope of the present invention is not to be limited to theparticular embodiments discussed. Thus, the embodiments shall beregarded as illustrative rather than restrictive, and it should beunderstood that variations may be made in those embodiments by workersskilled in the arts without departing from the scope of the presentinvention.

In addition, in methods that may be performed according to preferredembodiments herein and that may have been described above, theoperations have been described in selected typographical sequences.However, the sequences have been selected and so ordered fortypographical convenience and are not intended to imply any particularorder for performing the operations, except for those where a particularorder may be expressly set forth or where those of ordinary skill in theart may deem a particular order to be necessary.

In addition, all references cited above and below herein, as well as thebackground, invention summary, abstract and brief description of thedrawings, are all incorporated by reference into the detaileddescription of the preferred embodiments as disclosing alternativeembodiments.

The following are incorporated by reference: U.S. Pat. Nos. 7,587,085,7,587,068, 7,574,016, 7,565,030, 7,564,994, 7,558,408, 7,555,148,7,551,755, 7,551,754, 7,545,995, 7,515,740, 7,471,846, 7,469,071,7,469,055, 7,466,866, 7,460,695, 7,460,694, 7,440,593, 7,436,998,7,403,643, 7,352,394, 6,407,777, 7,269,292, 7,308,156, 7,315,631,7,336,821, 7,295,233, 6,571,003, 7,212,657, 7,039,222, 7,082,211,7,184,578, 7,187,788, 6,639,685, 6,628,842, 6,256,058, 5,579,063,6,480,300, 5,781,650, 7,362,368 and 5,978,519; and

U.S. published application nos. 2008/0175481, 2008/0013798,2008/0031498, 2005/0041121, 2007/0110305, 2006/0204110,PCT/US2006/021393, 2005/0068452, 2006/0120599, 2006/0098890,2006/0140455, 2006/0285754, 2008/0031498, 2007/0147820, 2007/0189748,2008/0037840, 2007/0269108, 2007/0201724, 2002/0081003, 2003/0198384,2006/0276698, 2004/0080631, 2008/0106615, 2006/0077261 and 2007/0071347;and

U.S. patent application Ser. Nos. 10/764,339, 11/861,854, 11/573,713,11/462,035, 12/042,335, 12/063,089, 11/761,647, 11/753,098, 12/038,777,12/043,025, 11/752,925, 11/767,412, 11/624,683, 60/829,127, 12/042,104,11/856,721, 11/936,085, 12/142,773, 60/914,962, 12/038,147, 11/861,257,12/026,484, 11/861,854, 61/024,551, 61/019,370, 61/023,946, 61/024,508,61/023,774, 61/023,855, 61/221,467, 61/221,425, 61/221,417, 61/091,700,61/182,625, 61/221,455, 11/319,766, 11/673,560, 12/485,316, 12/374,040,12/479,658, 12/479,593, 12/362,399, 12/191,304, 11/937,377, 12/038,147,12/055,958, 12/026,484, 12/554,258, 12/437,464, 12/042,104, 12/485,316,and 12/302,493.

What is claimed is:
 1. A digital image acquisition device including anoptical system having first and second lens systems for acquiringdigital images, and one or more processor-readable media having embodiedtherein processor-readable code for programming one or more processorsto perform the method comprising the following, not necessarily in theorder stated: superimposing on a representation of a first scene asubject locator symbol, the representation of the first scene acquiredthrough the first lens system of the digital image acquisition device;after superimposing the subject locator symbol, acquiring arepresentation of a second scene through a second lens system of thedigital image acquisition device; scaling at least part of therepresentation of the second scene to substantially the same size as thesubject locator symbol; and inserting the scaled part of therepresentation of the second scene into the representation of the firstscene at the position of the subject locator symbol to form a compositeimage, wherein the subject locator symbol is directly scalable andpositionable relative to the representation of the first scene inresponse to user input provided to the subject locator symbol, prior toacquiring the representation of the second scene.
 2. The digital imageacquisition device of claim 1, wherein the first lens system faces afirst direction relative to the device and the second lens system facesa second direction relative to the device, wherein the second directionis opposite to the first direction.
 3. The digital image acquisitiondevice of claim 1, wherein the representation of the first scenecomprises a series of preview images.
 4. The digital image acquisitiondevice of claim 1, wherein the representation of the first scenecomprises an image retrieved from memory.
 5. The digital imageacquisition device of claim 1, wherein the representation of the secondscene comprises a series of preview images.
 6. The digital imageacquisition device of claim 1, wherein the representation of the secondscene is retrieved from memory.
 7. The digital image acquisition deviceof claim 1, wherein the representation of the first scene and therepresentation of the second scene are captured and stored in long termpersistent memory at substantially the same time.
 8. The digital imageacquisition device of claim 1, wherein acquiring the representation ofthe second scene through the second lens system comprises entering apreview mode.
 9. The digital image acquisition device of claim 1,wherein the representation of the second scene includes an image of auser.
 10. The digital image acquisition device of claim 1, wherein thedigital image acquisition device is a cell phone.
 11. The digitalacquisition device of claim 1, wherein acquiring a representation of thesecond scene comprises: obtaining one or more preview images of thesecond scene, the preview images being associated with a first qualitymeasure; and acquiring the representation of the second scene, therepresentation being associated with a second quality measure greaterthan the first quality measure.
 12. The digital acquisition device ofclaim 11, wherein the first quality measure indicates at least a firstresolution, and wherein the second quality measure indicates at least asecond resolution.
 13. A method of forming a composite image in adigital image acquisition device, comprising the following acts, notnecessarily in the order stated: displaying on a screen of the imageacquisition device a representation of a first scene acquired through afirst lens system of the digital image acquisition device; superimposingon the representation of the first scene on the screen a subject locatorsymbol, wherein the subject locator symbol is directly scalable andpositionable relative to the representation of the first scene inresponse to user input provided to the subject locator symbol; aftersuperimposing, acquiring a representation of a second scene through asecond lens system; scaling at least a part of the representation of thesecond scene to substantially the same size as the subject locatorsymbol; and inserting the scaled part of the representation of thesecond scene into the representation of the first scene to form thecomposite image.
 14. The method of claim 13, wherein the first lenssystem faces a first direction relative to the device and the secondlens system faces a second direction relative to the device, wherein thesecond direction is opposite to the first direction.
 15. The method ofclaim 13, wherein the representation of the first scene comprises aseries of preview images.
 16. The method of claim 13, wherein therepresentation of the first scene comprises an image retrieved frommemory.
 17. The method of claim 13, wherein acquiring the representationof the second scene comprises acquiring a series of preview images. 18.The method of claim 13, wherein acquiring the representation of thesecond scene comprises capturing and storing an image.
 19. The method ofclaim 13, further comprising acquiring the representation of the firstscene before acquiring the representation of the second scene.
 20. Themethod of claim 13, further comprising displaying the representation ofthe first scene and the representation of the second scene on the screenat substantially the same time.
 21. The method of claim 13, furthercomprising capturing and storing the representation of the first sceneand the representation of the second scene at substantially the sametime.
 22. The method of claim 13, further comprising causing the digitalimage acquisition device to enter a user-composite mode.
 23. The methodof claim 13, further comprising acquiring the representation of thefirst scene from storage.
 24. The method of claim 13, further comprisingacquiring the representation of the second scene from storage.
 25. Themethod of claim 13, wherein the digital image acquisition device is acell phone.
 26. The method of claim 13, wherein the representation ofthe second scene includes an image of a user.
 27. The method of claim13, wherein acquiring the representation of the second scene comprisescausing the digital image acquisition device to enter a preview mode.28. The method of claim 27, further comprising, after causing thedigital image acquisition device to enter the preview mode, capturingand storing a composite image in long term persistent memory.
 29. Adigital image acquisition device including a screen, an optical systemhaving first and second lens systems for acquiring digital images, andone or more processor-readable media having embodied thereinprocessor-readable code for programming one or more processors toperform the method comprising the following, not necessarily in theorder stated: displaying on the screen of the image acquisition device arepresentation of a first scene acquired through the first lens system;superimposing on the representation of the first scene on the screen asubject locator symbol, wherein the subject locator symbol is directlyscalable and positionable relative to the representation of the firstscene in response to user input provided to the subject locator symbol;acquiring a representation of a second scene through the second lenssystem; scaling at least a part of the representation of the secondscene to substantially the same size as the subject locator symbol; andinserting the scaled part of the representation of the second scene intothe representation of the first scene to form a composite image.
 30. Thedigital image acquisition device of claim 29, wherein the subjectlocator symbol is directly scalable and positionable prior to acquiringthe representation of the second scene.